An Effective Training Method For Deep Convolutional Neural Network

We present a training method to speed up the training and improve the performance of deep convolutional neural networks (CNN). We propose a nonlinearity generator, which makes the deep CNN as a linear model in the initial state, and then introduces nonlinearity during the training procedure to improve the model capacity. We theoretically show that the mean shift problem in the neural network makes the training unstable, and the proposed method can partly solve this problem. The nonlinearity generator (NG) can be considered as a regularizer to make the feature map more discriminative than traditional methods. Experiments show that our method speeds up the convergence of the back propagation algorithm, enables larger learning rate and allows less careful initialization; it also improves the performance of CNN at negligible extra computational cost and can be jointly used with batch normalization to further improve the model performance. We train an extremely deep CNN with the proposed method easily without the extra training tricks.